Image process with spatial periodicity measure

ABSTRACT

An image manipulation process is controlled by a spatial periodicity measure formed for an image or an image block by measuring the sparseness of the two-dimensional spatial frequency spectrum of the image on a scale of zero to unity in which a two-dimensional spatial frequency spectrum having all equal values has a sparseness of zero and in which a two-dimensional spatial frequency spectrum having only one non-zero value has a sparseness of unity. Sparseness may be measured by allocating values of the spectrum to frequency bins and counting the number of bins that contain non-zero values; comparing values of the spectrum with a threshold and counting the number of values that exceed the threshold; or forming a function of the mean-square value and the mean value of the spectrum values.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. patentapplication Ser. No. 14/036,054, filed Sep. 25, 2013 which claims thebenefit of Great Britain Application No. GB 1217114.6, filed Sep. 25,2012, the entire disclosure of which is hereby incorporated byreference.

FIELD OF INVENTION

This invention concerns image processing and, in particular, thedetection of periodic structures.

BACKGROUND OF THE INVENTION

Spatially periodic structures, that is to say image features that repeatover at least part of the image area can cause problems for some imageprocessing tasks. (In the remainder of this specification periodicitymeans spatial periodicity unless another meaning is specificallystated.) Railings, empty seats in an Olympic stadium, and windows ofskyscrapers are typical periodic structures. An example of an imageprocessing task for which periodic structures are problematic is motioncompensated frame rate interpolation, and the difficulties of aparticular case will now be explained with reference to FIGS. 1 to 3.

FIG. 1 shows one-dimensional sections through two successive frames(101), (102) in a time sequence of sampled images, referred to as theprevious frame (101) and next frame (102). A set of adjacent pixelsalong a particular direction within a particular image region isrepresented by small circles in the Figure. The luminance values of thepixels are represented by the fill pattern of the circles, and, forsimplicity it is assumed that only three luminance valuesexist—represented by white fill, hatched fill, and black fill.

The spatial sequence of pixel values comprises a periodic structure witha period of six pixel pitches (103). This structure moves, over adistance of two pixel pitches, to a new position in the next frame(102). This is indicated by the motion vector (105) which is shownpointing from a pixel (106) in the previous frame to a matching pixel(107) in the next frame. An incorrect motion vector (108) is also shown,with a motion of four times the pixel pitch in the opposite direction.Though incorrect with regard to the actual motion of the structure, thissecond motion vector (108) represents an equally valid interpretation ofthe local information in the two images, demonstrated by the exactmatching of the values of pixels (106) and (109).

For some applications of motion estimation, incorrect motion vectorsarising from periodic structures do not pose a significant problem. Forexample, in compression involving motion compensated interframeprediction, the prediction resulting from such an incorrect vector maybe as good as that resulting from the correct vector. The onlydisadvantage in using the incorrect vector is that the motion vectorsmay be less consistent, so the cost of transmitting motion vectors mayincrease slightly.

For other applications, such as motion compensated frame rateinterpolation, incorrect motion vectors arising from periodic structuresmay significantly degrade the performance of the interpolation. In FIG.2, interpolation of an intermediate image (203) using correct motionvectors is shown. In this exemplary motion-compensated process, theinterpolated output frame (203) is derived from the combination of: aframe (205) that is a shifted version of the previous frame (201); and,a frame (207) that is a shifted version of the next frame (202). Forclarity the two contributing frames are shown in the Figure adjacent tothe position of the output frame (203), however their true positionscoincide with position of the output frame (203).

Pixels are projected from the previous image (201) according to acorrect forward vector (204) to form the forward projected image (205),and pixels are projected from the next image (202) according to acorrect backward vector (206) (which is equal in size and opposite indirection to the forward vector) to form the backward projected image(207). The average of the two (in this case identical) projected imagesis used to produce the interpolated output image (203).

We now consider an example, illustrated in FIG. 3, where one of themotion vectors, in this case the backward vector (306), is incorrectbecause of the ambiguity arising from the periodic structure. Thebackward projected frame (307) now has a spatial phase difference of180° (three pixel pitches) with respect to the forwards projection, withthe result that the two contributions (305), (307) to the interpolatedresult (303) are misaligned and make an incorrect interpolated image(303). The resulting interpolated moving sequence would have significantflicker and motion judder.

Several methods for detecting and for correcting for periodic structuresin images have been proposed. In U.S. Patent 2010/008423, Namboodiri etal disclose a method for periodic structure detection in whichsignificant peaks are sought in the two-dimensional frequency-domainrepresentation of blocks in a picture. In “Detecting PeriodicStructures”, Orwell et al [Orwell, J. M. and Boyce, J. F. ‘DetectingPeriodic Structures’, Proc. Fourteenth International Conference onPattern Recognition, 1998, pp. 714-716] disclose a method for periodicstructure detection based on feature detection followed by anautocorrelation process on the detected features. In “Periodicity,directionality and randomness: Wold features for image modeling andretrieval” Liu and Picard [Liu, F. and Picard, R. W. ‘Periodicity,directionality and randomness: Wold features for image modeling andretrieval’, IEEE Trans. on PAMI, vol. 18, no. 7, July 1996, pp. 722-733]disclose a method for periodic structure detection based on imageautocovariance functions. In U.S. Pat. No. 5,793,430, Knee et aldisclose a method based on analysing a line of block-based displacedframe differences in a block matching motion estimator.

The prior-art methods are either very complex to implement, fail todetect periodic structures with multiple dominant frequency components,or give “false alarms” on non-periodic structures such as straightedges. The present invention addresses some or all of these deficienciesof the prior art.

SUMMARY OF THE INVENTION

The inventor has observed that an important indicator of the presence ofperiodic structures in an image is the sparsity of the representation ofthe image in the two-dimensional frequency domain, and has developed amethod and apparatus for periodic structure detection that exploits thatobservation.

The invention consists in an image manipulation process that depends ona spatial periodicity measure for image data that is a function of thesparseness of the two-dimensional spatial frequency spectrum of theprocessed image data.

In a preferred embodiment, at least one two-dimensional block of imagedata is processed in a two-dimensional Fourier transform, and aperiodicity measure for the block(s) is evaluated from the portion ofthe transformed image data that represents AC components of thespectrum.

Preferably the said sparseness is a function of the mean-square value ofthe transformed image data for the said two-dimensional block and themean value of the transformed image data for that two-dimensional block.

Advantageously the sparseness is a linear function of the mean-squarevalue of the transformed image data and the square of the mean value ofthe transformed image data.

The sparseness may further depend on the mean-square value of one ormore transform output values corresponding to low spatial frequencies.

BRIEF DESCRIPTION OF THE DRAWINGS

Examples of the invention will now be described with reference to thedrawings in which:

FIG. 1 is a diagram showing a correct and an incorrect motion vectormeasured between two frames of an image sequence containing a periodicstructure;

FIG. 2 is a diagram showing motion compensated temporal interpolationusing correct motion vectors between two frames of a sequence containinga periodic structure according to the prior art;

FIG. 3 is a diagram showing motion compensated temporal interpolationusing incorrect motion vectors between two frames of a sequencecontaining a periodic structure according to the prior art;

FIG. 4 is a block diagram of a preferred embodiment of a periodicstructure detection system according to the invention;

FIG. 5 is a block diagram of interpolation apparatus according to theinvention.

DETAILED DESCRIPTION OF THE INVENTION

In an exemplary image manipulation process according to the invention,an image to be processed using motion compensation is divided into anumber of regions, or ‘blocks’, and a periodicity measure is derived foreach block. These periodicity values are used to control the operationof the motion compensation process, so that more weight is given to thezero motion vector when processing blocks having high periodicitymeasures. Thus these blocks are less likely to be shifted by,potentially unreliable, non-zero vectors.

A suitable method of deriving periodicity values for the blocks of animage will now be described with reference to FIG. 4. Incoming luminancedata (401) that corresponds to the pixels of an image is passed to ablock formatter (402) where it is formatted into rectangular blocks ofpixels (403). Typical block dimensions are 32×32 pixels, and the blocksmay overlap. Each block of pixel values is processed as described belowto obtain a periodicity measure (417) for the block.

The block data is passed to a two-dimensional frequency transform suchas an FFT (404) to produce a transformed set of data values (405) thatare a function of the magnitudes of the spatial frequency componentsthat comprise the spatial frequency spectrum of the block. For a 32×32pixel block, there will be 1,024 values which can conveniently bedenoted F_(i,j) and arranged so that:

-   F_(0,0) is the magnitude of the DC component;-   F_(15,0) is the magnitude of the highest positive horizontal    frequency component;-   F_(−16,0) is the magnitude of the highest negative horizontal    frequency component;-   F_(0,15) is the magnitude of the highest positive vertical frequency    component;-   F_(0,−16) is the magnitude of the highest negative vertical    frequency component; etc.

The values F_(i,j) are subjected to three processes whose outputs arecombined in a combination block (416) that outputs a periodicity measure(417) for each block. The magnitude of the DC component F_(0,0) is notused in these subsequent processes, which thus operate on the ACspectrum of the block.

In a first process the magnitudes of the non-DC components are squared(406) and averaged (412) to produce a mean-square sum (413) of thespectrum of the block. This represents mean-square spectral energy forthe block and will be denoted as E_(ms) so that:

E _(ms)=(ΣF _(i,j) ²)÷N  [1]

-   where the summation includes all spectral components F_(i,j)    excluding F_(0,0); and,-   N is the total number of spectral components (1,024 for a 32×32    pixel block).

In a second process the magnitudes of the non-DC components are averaged(410) to produce an average spectral volume (411) for the block. Thiswill be denoted S_(avg) so that:

S _(avg)=(ΣF _(i,j))÷N  [2]

-   where the summation includes all spectral components F_(i,j)    excluding F_(0,0); and,-   N is the total number of spectral components (1,024 for a 32×32    pixel block).

In a third process the squares of the lowest-frequency spectralcomponents, including horizontal, vertical and diagonal components, areaveraged (414) to produce a mean-square sum of the low-frequencyspectrum (415) of the block. This will be designated L_(ms) so that:

L _(ms)=(F _(−1,−1) ² +F _(0,−1) ² +F _(1,−1) ² +F _(−1,0) ² +F _(1,0) ²+F _(−1,1) ² +F _(0,1) ² +F _(1,1) ²)÷N  [3]

-   where: F_(−1,−1) is the magnitude of the lowest negative    backward-diagonal frequency component;-   F_(0,−1) is the magnitude of the lowest negative vertical frequency    component;-   F_(1,−1) is the magnitude of the lowest negative forward-diagonal    frequency component;-   F_(−1,0) is the magnitude of the lowest negative horizontal    frequency component;-   F_(1,0) is the magnitude of the lowest positive horizontal frequency    component;-   F_(−1,1) is the magnitude of the lowest positive forward-diagonal    frequency component;-   F_(0,1) is the magnitude of the lowest positive vertical frequency    component;-   F_(1,1) is the magnitude of the lowest positive backward-diagonal    frequency component; and,-   N is the total number of spectral components (1,024 for a 32×32    pixel block).

Finally, the mean-square spectral energy E_(ms) (413), the averagespectral volume S_(avg) (411), and the low-frequency mean-squarespectral energy L_(ms) (415) are combined in a mathematical function(416) to produce a periodicity measure (417) for the block.

The principle by which the invention works is that blocks with asignificant amount of periodic content, will have spatial frequencyspectra that are “sparse”. If the spectrum of the block is dividedevenly into frequency regions, for example the frequency “bins” of aFourier transform, the majority of the energy will be concentrated in aminority of the total number of regions. Sparseness may also be regardedas the extent to which the distribution of values departs from theuniform distribution having equal values. Thus, on a normalised scale, adistribution which had only one non-zero value might have a sparsenessmeasure of unity whilst a uniform distribution might have a sparsenessvalue of zero. For further discussion of the concept of “sparseness”,reference is directed to a paper by Patrik O. Hoyer entitled“Non-negative matrix factorization with sparseness constraints” inJournal of Machine Learning Research 5 (2004) pp 1457-1469.

One general approach is to compare mean-square value of the transformedimage data with the square of the mean value of the transformed imagedata. For example, a periodicity measure might be taken as thedifference between (or other linear function of) the mean-square valueof the transformed image data and the square of the mean value of thetransformed image data. For example a periodicity measure ψ may be givenby:

ψ=E _(ms) −S _(avg) ²  [4]

The square of the average spectral volume of a sparse spectrum will berelatively low, as compared to the mean-square energy, thus leading to ahigh periodicity measure. As an example, consider two 32×32 blocks, thefirst having F_(i,j) values of unity throughout the block, and thesecond having a more sparse spectrum with 256 F_(i,j) values of 2, andvalues of 0 for the remainder. The average spectral volume for the firstblock will be unity, and its square will also be 1. The second block hasan average spectral volume of ½, so the square will be ¼. In both casesthe mean-square energy will be unity.

If a significant proportion of the block energy is at very lowfrequencies, there is a high probability that the block contains ahigh-contrast edge or a large luminance ramp. As these are not periodicstructures, the periodicity measure may be reduced when low-frequencyspectral components are detected. For example:

ψ=γE _(ms) −αS _(avg) ² −βL _(ms)  [5]

If 8-bit representation is used for the original luminance samples, thensuitable values for the parameters are γ=1, α=2, β=3, δ=96.

A more particular example of the function (416) will now be given. Theperiodicity measure ψ is given by:

$\begin{matrix}{\psi = {\min \left\{ {1,{\max \left\{ {0,\frac{{\gamma \; E_{ms}} - {\alpha \; S_{avg}^{2}} - {\beta \; L_{ms}}}{\max \left( {\delta,E_{ms}} \right)}} \right\}}} \right\}}} & \lbrack 6\rbrack\end{matrix}$

The maximum and minimum functions, and the division by E_(ms), inequation [6] above provide “soft” normalization of the value of theperiodicity measure, limiting its value to the range from zero to unity.Other techniques for providing normalisation of the approaches ofequation [4] or equation [5] will occur to the skilled reader.

In another approach, a measure of sparseness is provided by counting thenumber of frequency bins that contain non-zero values or values above adefined threshold. That defined threshold might be a defined proportion(for example one quarter, one third or one half) of the mean value overthe block. The mean of spatial frequency values may be replaced by otherrepresentative values such as the median or mode. In this approach, thesmaller the number of values above the threshold, the greater is thesparseness.

The resulting periodicity value v may be used directly in furtherprocessing, for example to control bias values in a motion vectorselection process, or it may be subjected to a threshold to give abinary “periodic/non-periodic” decision. A suitable threshold value is0.5.

The invention can be implemented in many ways. For example theprocessing may be carried out in real time on streaming data, or at anarbitrary rate on stored data. Different transforms may be used toconvert the image data from the sampled pixel value domain to thespatial frequency domain.

The image may be down-sampled or up-sampled prior to evaluation of itsspectral sparseness and the processing may be applied to a single blockcomprising the entire image or a part of the image.

In the above-described example luminance values for pixels were used.However other numerical parameters of pixels, such as colour-separationvalues, or colour difference values could be used.

The skilled person will appreciate that there are many motioncompensated video processes that can benefit from control by one or moreperiodicity measures determined according to the invention. Examplesinclude interpolation of new images within a sequence of images, datacompression, and noise reduction.

Referring to FIG. 5, interpolation apparatus is shown in which inputimages are presented to a motion compensated interpolator and also to aspatial periodicity measurement block and to a motion measurement block.

The spatial periodicity measurement block serves to provide for eachinput image or for each block of an input image a spatial periodicitymeasure using any of the techniques described above. The motionmeasurement block may take any of a number of well-known forms. Thespatial periodicity values are used by the interpolator to control theoperation of the motion compensation process, so that—for example—moreweight is given to the zero motion vector when processing blocks havinghigh spatial periodicity measures. Thus these blocks are less likely tobe shifted in a motion compensation process by, potentially unreliable,non-zero vectors.

The skilled man will understand that this invention has been describedby way of example only and that a wide variety of modifications andalternative are possible within the scope of the appended claims and anyequivalents thereof.

What is claimed:
 1. A non-transitory computer-readable medium includinginstructions configured to cause a computer system to implement a methodcomprising the steps of: forming a spatial periodicity measure for atleast part of an image; and controlling an image manipulation process independence on said spatial periodicity measure; wherein the step offorming a spatial periodicity measure comprises: measuring thesparseness of the two-dimensional spatial frequency spectrum of theimage on a scale of zero to unity in which a two-dimensional spatialfrequency spectrum having all equal values has a sparseness of zero andin which a two-dimensional spatial frequency spectrum having only onenon-zero value has a sparseness of unity; and deriving a spatialperiodicity measure data that is a function of said sparseness of thetwo-dimensional spatial frequency spectrum.
 2. The non-transitorycomputer-readable medium of claim 1 in which at least onetwo-dimensional block of image data is processed in a two-dimensionalspatial to frequency domain transform to form transformed image data. 3.The non-transitory computer-readable medium of claim 2 in which the saidspatial periodicity measure is normalized by division by a function ofthe mean-square transformed image data for the said two-dimensionalblock.
 4. The non-transitory computer-readable medium of claim 2 inwhich the said sparseness is a function of the mean-square value of thetransformed image data for the said two-dimensional block and the meanvalue of the transformed image data for that two-dimensional block. 5.The non-transitory computer-readable medium of claim 4 in which thesparseness is a linear function of the mean-square value of thetransformed image data and the square of the mean value of thetransformed image data.
 6. The non-transitory computer-readable mediumof claim 4 in which the sparseness further depends on the mean-squarevalue of one or more transform output values corresponding to lowspatial frequencies.
 7. The non-transitory computer-readable medium ofclaim 1 in which said sparseness is measured by counting the number ofspatial frequency values in the two-dimensional spatial frequencyspectrum that exceed a threshold value.
 8. The non-transitorycomputer-readable medium of claim 7 in which said threshold comprises arepresentative value for the spatial frequency values in thetwo-dimensional spatial frequency spectrum.
 9. The non-transitorycomputer-readable medium of claim 1 in which measuring the sparseness ofthe two-dimensional spatial frequency spectrum of the image utilizes atechnique selected from the group of: allocating values of the spectrumto frequency bins and counting the number of bins that contain non-zerovalues; comparing values of the spectrum with a threshold and countingthe number of values that exceed the threshold; comparing values of thespectrum with a representative value for the spectrum and counting thenumber of values that exceed the representative value; and forming afunction of the mean-square value of the spectrum values and the meanvalue of the spectrum values.
 10. The non-transitory computer-readablemedium of claim 1 in which the said image manipulation process comprisesthe interpolation of at least one new image within a sequence of images.11. The non-transitory computer-readable medium of claim 10, in whichsaid interpolation comprises motion compensated interpolation and inwhich said motion compensation is controlled in dependence on saidspatial periodicity measure.
 12. A non-transitory computer-readablemedium including instructions configured to cause a computer system toimplement a motion compensated image manipulation process comprising thesteps of: forming a spatial periodicity measure for at least part of animage; and controlling motion compensation of said motion compensatedimage manipulation process in dependence on said spatial periodicitymeasure; wherein the step of forming a spatial periodicity measurecomprises: measuring the sparseness of the two-dimensional spatialfrequency spectrum of the image utilizing a technique selected from thegroup of: allocating values of the spectrum to frequency bins andcounting the number of bins that contain non-zero values; comparingvalues of the spectrum with a threshold and counting the number ofvalues that exceed the threshold; comparing values of the spectrum witha representative value for the spectrum and counting the number ofvalues that exceed the representative value; and forming a function ofthe mean-square value of the spectrum values and the mean value of thespectrum values; and deriving a spatial periodicity measure data that isa function of said sparseness of the two-dimensional spatial frequencyspectrum.
 13. The non-transitory computer-readable medium of claim 12 inwhich at least one two-dimensional block of image data is processed in atwo-dimensional spatial to frequency domain transform to formtransformed image data.
 14. The non-transitory computer-readable mediumof claim 13 in which the said spatial periodicity measure is normalizedby division by a function of the mean-square transformed image data forthe said two-dimensional block.
 15. The non-transitory computer-readablemedium of claim 12 in which the said image manipulation processcomprises the interpolation of at least one new image within a sequenceof images.
 16. The non-transitory computer-readable medium of claim 12where said measure of the sparseness of the two-dimensional spatialfrequency spectrum of the image has a scale of zero to unity in which atwo-dimensional spatial frequency spectrum having all equal values has asparseness of zero and in which a two-dimensional spatial frequencyspectrum having only one non-zero value has a sparseness of unity.
 17. Amethod of controlling a motion compensated image manipulation process ina processor, the method comprising the steps of: forming a spatialperiodicity measure for at least part of an image; and controllingmotion compensation of said image manipulation process in dependence onsaid spatial periodicity measure; wherein the step of forming a spatialperiodicity measure comprises: measuring the sparseness of thetwo-dimensional spatial frequency spectrum of the image utilizing atechnique selected from the group of: allocating values of the spectrumto frequency bins and counting the number of bins that contain non-zerovalues; comparing values of the spectrum with a threshold and countingthe number of values that exceed the threshold; comparing values of thespectrum with a representative value for the spectrum and counting thenumber of values that exceed the representative value; and forming afunction of the mean-square value of the spectrum values and the meanvalue of the spectrum values; and deriving a spatial periodicity measuredata that is a function of said sparseness of the two-dimensionalspatial frequency spectrum.
 18. The method of claim 17 in which at leastone two-dimensional block of image data is processed in atwo-dimensional spatial to frequency domain transform to formtransformed image data.
 19. The method of claim 17 in which the saidimage manipulation process comprises the interpolation of at least onenew image within a sequence of images.
 20. The method of claim 17 wheresaid measure of the sparseness of the two-dimensional spatial frequencyspectrum of the image has a scale of zero to unity in which atwo-dimensional spatial frequency spectrum having all equal values has asparseness of zero and in which a two-dimensional spatial frequencyspectrum having only one non-zero value has a sparseness of unity.